{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout\n",
    "\n",
    ":label:`sec_dropout`\n",
    "\n",
    "\n",
    "In :numref:`sec_weight_decay`,\n",
    "we introduced the classical approach\n",
    "to regularizing statistical models \n",
    "by penalizing the $\\ell_2$ norm of the weights.\n",
    "In probabilistic terms, we could justify this technique\n",
    "by arguing that we have assumed a prior belief\n",
    "that weights take values from \n",
    "a Gaussian distribution with mean $0$.\n",
    "More intuitively, we might argue\n",
    "that we encouraged the model to spread out its weights\n",
    "among many features rather than depending too much\n",
    "on a small number of potentially spurious associations.\n",
    "\n",
    "## Overfitting Revisited\n",
    "\n",
    "Faced with more features than examples, \n",
    "linear models tend to overfit.\n",
    "But given more examples than features,\n",
    "we can generally count on linear models not to overfit.\n",
    "Unfortunately, the reliability with which\n",
    "linear models generalize comes at a cost:\n",
    "Naively applied, linear models do not take \n",
    "into account interactions among features.\n",
    "For every feature, a linear model must assign\n",
    "either a positive or a negative weight, ignoring context.\n",
    "\n",
    "In traditional texts, this fundamental tension \n",
    "between generalizability and flexibility\n",
    "is described as the *bias-variance tradeoff*.\n",
    "Linear models have high bias\n",
    "(they can only represent a small class of functions),\n",
    "but low variance (they give similar results\n",
    "across different random samples of the data).\n",
    "\n",
    "Deep neural networks inhabit the opposite \n",
    "end of the bias-variance spectrum.\n",
    "Unlike linear models, neural networks\n",
    "are not confined to looking at each feature individually.\n",
    "They can learn interactions among groups of features.\n",
    "For example, they might infer that \n",
    "“Nigeria” and “Western Union” appearing \n",
    "together in an email indicates spam\n",
    "but that separately they do not.\n",
    "\n",
    "Even when we have far more examples than features,\n",
    "deep neural networks are capable of overfitting.\n",
    "In 2017, a group of researchers demonstrated\n",
    "the extreme flexibility of neural networks\n",
    "by training deep nets on randomly-labeled images.\n",
    "Despite the absence of any true pattern\n",
    "linking the inputs to the outputs,\n",
    "they found that the neural network optimized by SGD\n",
    "could label every image in the training set perfectly.\n",
    "\n",
    "Consider what this means.\n",
    "If the labels are assigned uniformly\n",
    "at random and there are 10 classes,\n",
    "then no classifier can do better \n",
    "than 10% accuracy on holdout data.\n",
    "The generalization gap here is a whopping 90%.\n",
    "If our models are so expressive that they \n",
    "can overfit this badly, then when should\n",
    "we expect them not to overfit?\n",
    "The mathematical foundations for \n",
    "the puzzling generalization properties\n",
    "of deep networks remain open research questions,\n",
    "and we encourage the theoretically-oriented \n",
    "reader to dig deeper into the topic.\n",
    "For now, we turn to the more terrestrial investigation of \n",
    "practical tools that tend (empirically)\n",
    "to improve the generalization of deep nets.\n",
    "\n",
    "\n",
    "## Robustness through Perturbations\n",
    "\n",
    "Let us think briefly about what we \n",
    "expect from a good predictive model.\n",
    "We want it to peform well on unseen data.\n",
    "Classical generalization theory\n",
    "suggests that to close the gap between\n",
    "train and test performance, \n",
    "we should aim for a *simple* model.\n",
    "Simplicity can come in the form\n",
    "of a small number of dimensions.\n",
    "We explored this when discussing the\n",
    "monomial basis functions of linear models\n",
    ":numref:`sec_model_selection`.\n",
    "Additionally, as we saw when discussing weight decay \n",
    "($\\ell_2$ regularization) :numref:`sec_weight_decay`,\n",
    "the (inverse) norm of the parameters \n",
    "represents another useful measure of simplicity.\n",
    "Another useful notion of simplicity is smoothness,\n",
    "i.e., that the function should not be sensitive\n",
    "to small changes to its inputs.\n",
    "For instance, when we classify images,\n",
    "we would expect that adding some random noise\n",
    "to the pixels should be mostly harmless.\n",
    "\n",
    "In 1995, Christopher Bishop formalized\n",
    "this idea when he proved that training with input noise \n",
    "is equivalent to Tikhonov regularization :cite:`Bishop.1995`.\n",
    "This work drew a clear mathematical connection\n",
    "between the requirement that a function be smooth (and thus simple),\n",
    "and the requirement that it be resilient \n",
    "to perturbations in the input.\n",
    "\n",
    "Then, in 2014, Srivastava et al. :cite:`Srivastava.Hinton.Krizhevsky.ea.2014`\n",
    "developed a clever idea for how to apply Bishop's idea\n",
    "to the *internal* layers of the network, too.\n",
    "Namely, they proposed to inject noise \n",
    "into each layer of the network\n",
    "before calculating the subsequent layer during training.\n",
    "They realized that when training \n",
    "a deep network with many layers,\n",
    "injecting noise enforces smoothness just on the input-output mapping.\n",
    "\n",
    "Their idea, called *dropout*, involves \n",
    "injecting noise while computing \n",
    "each internal layer during forward propagation,\n",
    "and it has become a standard technique\n",
    "for training neural networks.\n",
    "The method is called *dropout* because we literally\n",
    "*drop out* some neurons during training.\n",
    "Throughout training, on each iteration,\n",
    "standard dropout consists of zeroing out \n",
    "some fraction (typically 50%) of the nodes in each layer\n",
    "before calculating the subsequent layer.\n",
    "\n",
    "To be clear, we are imposing \n",
    "our own narrative with the link to Bishop.\n",
    "The original paper on dropout\n",
    "offers intuition through a surprising \n",
    "analogy to sexual reproduction.\n",
    "The authors argue that neural network overfitting\n",
    "is characterized by a state in which \n",
    "each layer relies on a specifc \n",
    "pattern of activations in the previous layer,\n",
    "calling this condition *co-adaptation*.\n",
    "Dropout, they claim, breaks up co-adaptation\n",
    "just as sexual reproduction is argued to \n",
    "break up co-adapted genes. \n",
    "\n",
    "The key challenge then is *how* to inject this noise.\n",
    "One idea is to inject the noise in an *unbiased* manner\n",
    "so that the expected value of each layer---while fixing \n",
    "the others---equals to the value it would have taken absent noise.\n",
    "\n",
    "In Bishop's work, he added Gaussian noise \n",
    "to the inputs to a linear model:\n",
    "At each training iteration, he added noise\n",
    "sampled from a distribution with mean zero\n",
    "$\\epsilon \\sim \\mathcal{N}(0,\\sigma^2)$ to the input $\\mathbf{x}$,\n",
    "yielding a perturbed point $\\mathbf{x}' = \\mathbf{x} + \\epsilon$.\n",
    "In expectation, $E[\\mathbf{x}'] = \\mathbf{x}$.\n",
    "\n",
    "In standard dropout regularization,\n",
    "one debiases each layer by normalizing \n",
    "by the fraction of nodes that were retained (not dropped out).\n",
    "In other words, dropout with *dropout probability* $p$ \n",
    "is applied as follows:\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "h' =\n",
    "\\begin{cases}\n",
    "    0 & \\text{ with probability } p \\\\\n",
    "    \\frac{h}{1-p} & \\text{ otherwise}\n",
    "\\end{cases}\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "By design, the expectation remains unchanged, i.e., $E[h'] = h$.\n",
    "Intermediate activations $h$ are replaced by\n",
    "a random variable $h'$ with matching expectation.\n",
    "\n",
    "\n",
    "\n",
    "## Dropout in Practice\n",
    "\n",
    "Recall the multilayer perceptron (:numref:`sec_mlp`) \n",
    "with a hidden layer and 5 hidden units. \n",
    "Its architecture is given by\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "    \\mathbf{h} & = \\sigma(\\mathbf{W}_1 \\mathbf{x} + \\mathbf{b}_1), \\\\\n",
    "    \\mathbf{o} & = \\mathbf{W}_2 \\mathbf{h} + \\mathbf{b}_2, \\\\\n",
    "    \\hat{\\mathbf{y}} & = \\mathrm{softmax}(\\mathbf{o}).\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "When we apply dropout to a hidden layer,\n",
    "zeroing out each hidden unit with probability $p$,\n",
    "the result can be viewed as a network \n",
    "containing only a subset of the original neurons.\n",
    "In :numref:`fig_dropout2`, $h_2$ and $h_5$ are removed.\n",
    "Consequently, the calculation of $y$ \n",
    "no longer depends on $h_2$ and $h_5$\n",
    "and their respective gradient also vanishes \n",
    "when performing backprop.\n",
    "In this way, the calculation of the output layer\n",
    "cannot be overly dependent on any \n",
    "one element of $h_1, \\ldots, h_5$.\n",
    "\n",
    "![MLP before and after dropout](https://raw.githubusercontent.com/d2l-ai/d2l-en/master/img/dropout2.svg)\n",
    "\n",
    ":label:`fig_dropout2`\n",
    "\n",
    "\n",
    "Typically, ***we disable dropout at test time***.\n",
    "Given a trained model and a new example,\n",
    "we do not drop out any nodes \n",
    "(and thus do not need to normalize).\n",
    "However, there are some exceptions:\n",
    "some researchers use dropout at test time as a heuristic \n",
    "for estimating the *uncertainty* of neural network predictions:\n",
    "if the predictions agree across many different dropout masks,\n",
    "then we might say that the network is more confident.\n",
    "For now we will put off uncertainty estimation \n",
    "for subsequent chapters and volumes.\n",
    "\n",
    "\n",
    "## Implementation from Scratch\n",
    "\n",
    "To implement the dropout function for a single layer,\n",
    "we must draw as many samples \n",
    "from a Bernoulli (binary) random variable\n",
    "as our layer has dimensions, \n",
    "where the random variable takes value $1$ (keep) \n",
    "with probability $1-p$ and $0$ (drop) with probability $p$.\n",
    "One easy way to implement this is to first draw samples\n",
    "from the uniform distribution $U[0, 1]$.\n",
    "Then we can keep those nodes for which the corresponding\n",
    "sample is greater than $p$, dropping the rest.\n",
    "\n",
    "In the following code, we implement a `dropoutLayer()` function\n",
    "that drops out the elements in the `NDArray` input `X`\n",
    "with probability `dropout`,\n",
    "rescaling the remainder as described above\n",
    "(dividing the survivors by `1.0-dropout`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "// %mavenRepo snapshots https://oss.sonatype.org/content/repositories/snapshots/\n",
    "\n",
    "%maven ai.djl:api:0.8.0\n",
    "%maven ai.djl:basicdataset:0.8.0\n",
    "%maven org.slf4j:slf4j-api:1.7.26\n",
    "%maven org.slf4j:slf4j-simple:1.7.26\n",
    "    \n",
    "%maven ai.djl.mxnet:mxnet-engine:0.8.0\n",
    "%maven ai.djl.mxnet:mxnet-native-auto:1.7.0-backport"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%loadFromPOM\n",
    "<dependency>\n",
    "    <groupId>tech.tablesaw</groupId>\n",
    "    <artifactId>tablesaw-jsplot</artifactId>\n",
    "    <version>0.30.4</version>\n",
    "</dependency>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load ../utils/plot-utils.ipynb\n",
    "%load ../utils/DataPoints.java\n",
    "%load ../utils/Training.java\n",
    "%load ../utils/Accumulator.java"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import java.nio.file.*;\n",
    "\n",
    "import ai.djl.Device;\n",
    "import ai.djl.*;\n",
    "import ai.djl.engine.Engine;\n",
    "import ai.djl.metric.*;\n",
    "import ai.djl.ndarray.*;\n",
    "import ai.djl.ndarray.types.*;\n",
    "import ai.djl.ndarray.index.*;\n",
    "import ai.djl.nn.*;\n",
    "import ai.djl.nn.core.*;\n",
    "import ai.djl.training.*;\n",
    "import ai.djl.training.initializer.*;\n",
    "import ai.djl.training.loss.*;\n",
    "import ai.djl.training.listener.*;\n",
    "import ai.djl.training.evaluator.*;\n",
    "import ai.djl.training.optimizer.*;\n",
    "import ai.djl.training.tracker.*;\n",
    "import ai.djl.training.dataset.*;\n",
    "import ai.djl.util.*;\n",
    "import java.util.Random;\n",
    "import ai.djl.training.listener.TrainingListener;\n",
    "import ai.djl.basicdataset.FashionMnist;\n",
    "import ai.djl.training.dataset.Dataset;\n",
    "import ai.djl.nn.norm.Dropout;\n",
    "\n",
    "import tech.tablesaw.api.*;\n",
    "import tech.tablesaw.plotly.api.*;\n",
    "import tech.tablesaw.plotly.components.*;\n",
    "import tech.tablesaw.plotly.Plot;\n",
    "import tech.tablesaw.plotly.components.Figure;\n",
    "import org.apache.commons.lang3.ArrayUtils;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can test out the `dropoutLayer()` function on a few examples.\n",
    "In the following lines of code, \n",
    "we pass our input `X` through the dropout operation,\n",
    "with probabilities 0, 0.5, and 1, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "NDManager manager = NDManager.newBaseManager();\n",
    "\n",
    "public NDArray dropoutLayer(NDArray X, float dropout) {\n",
    "    // In this case, all elements are dropped out\n",
    "    if (dropout == 1.0f) {\n",
    "        return manager.zeros(X.getShape());\n",
    "    }\n",
    "    // In this case, all elements are kept\n",
    "    if (dropout == 0f) {\n",
    "        return X;\n",
    "    }\n",
    "\n",
    "    NDArray mask = manager.randomUniform(0f, 1.0f, X.getShape()).gt(dropout);\n",
    "    return mask.toType(DataType.FLOAT32, false).mul(X).div(1.0f - dropout);\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "2"
    }
   },
   "outputs": [],
   "source": [
    "NDArray X = manager.arange(16f).reshape(2, 8);\n",
    "System.out.println(dropoutLayer(X, 0));\n",
    "System.out.println(dropoutLayer(X, 0.5f));\n",
    "System.out.println(dropoutLayer(X, 1.0f));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining Model Parameters\n",
    "\n",
    "Again, we work with the Fashion-MNIST dataset\n",
    "introduced in :numref:`sec_softmax_scratch`.\n",
    "We define a multilayer perceptron with \n",
    "two hidden layers containing 256 outputs each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "3"
    }
   },
   "outputs": [],
   "source": [
    "int numInputs = 784;\n",
    "int numOutputs = 10;\n",
    "int numHiddens1 = 256;\n",
    "int numHiddens2 = 256;\n",
    "\n",
    "NDArray W1 = manager.randomNormal(0, 0.01f, new Shape(numInputs, numHiddens1), DataType.FLOAT32, Device.defaultDevice());\n",
    "NDArray b1 = manager.zeros(new Shape(numHiddens1));\n",
    "NDArray W2 = manager.randomNormal(0, 0.01f, new Shape(numHiddens1, numHiddens2), DataType.FLOAT32, Device.defaultDevice());\n",
    "NDArray b2 = manager.zeros(new Shape(numHiddens2));\n",
    "NDArray W3 = manager.randomNormal(0, 0.01f, new Shape(numHiddens2, numOutputs), DataType.FLOAT32, Device.defaultDevice());\n",
    "NDArray b3 = manager.zeros(new Shape(numOutputs));\n",
    "\n",
    "\n",
    "NDList params = new NDList(W1, b1, W2, b2, W3, b3);\n",
    "\n",
    "for (NDArray param : params) {\n",
    "    param.attachGradient();\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining the Model\n",
    "\n",
    "The model below applies dropout to the output \n",
    "of each hidden layer (following the activation function).\n",
    "We can set dropout probabilities for each layer separately.\n",
    "A common trend is to set\n",
    "a lower dropout probability closer to the input layer.\n",
    "Below we set it to 0.2 and 0.5 for the first \n",
    "and second hidden layer respectively.\n",
    "By using the `isTraining` boolean variable described in :numref:`sec_autograd`,\n",
    "we can ensure that dropout is only active during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "4"
    }
   },
   "outputs": [],
   "source": [
    "float dropout1 = 0.2f;\n",
    "float dropout2 = 0.5f;\n",
    "\n",
    "public NDArray net(NDArray X, boolean isTraining) {\n",
    "\n",
    "    X = X.reshape(-1, numInputs);\n",
    "    NDArray H1 = Activation.relu(X.dot(W1).add(b1));\n",
    "\n",
    "    if (isTraining) {\n",
    "        H1 = dropoutLayer(H1, dropout1);\n",
    "    }\n",
    "\n",
    "    NDArray H2 = Activation.relu(H1.dot(W2).add(b2));\n",
    "    if (isTraining) {\n",
    "        H2 = dropoutLayer(H2, dropout2);\n",
    "    }\n",
    "\n",
    "    return H2.dot(W3).add(b3);\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training and Testing\n",
    "\n",
    "This is similar to the training and testing of multilayer perceptrons described previously."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "int numEpochs = 10;\n",
    "float lr = 0.5f;\n",
    "int batchSize = 256;\n",
    "\n",
    "double[] trainLoss;\n",
    "double[] testAccuracy;\n",
    "double[] epochCount;\n",
    "double[] trainAccuracy;\n",
    "\n",
    "trainLoss = new double[numEpochs];\n",
    "trainAccuracy = new double[numEpochs];\n",
    "testAccuracy = new double[numEpochs];\n",
    "epochCount = new double[numEpochs];\n",
    "\n",
    "Loss loss = new SoftmaxCrossEntropyLoss();\n",
    "\n",
    "FashionMnist trainIter = FashionMnist.builder()\n",
    "                            .optUsage(Dataset.Usage.TRAIN)\n",
    "                            .setSampling(batchSize, true)\n",
    "                            .build();\n",
    "\n",
    "\n",
    "FashionMnist testIter = FashionMnist.builder()\n",
    "                            .optUsage(Dataset.Usage.TEST)\n",
    "                            .setSampling(batchSize, true)\n",
    "                            .build();\n",
    "                            \n",
    "trainIter.prepare();\n",
    "testIter.prepare();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "5"
    }
   },
   "outputs": [],
   "source": [
    "float epochLoss = 0f;\n",
    "float accuracyVal = 0f;\n",
    "\n",
    "for (int epoch = 1; epoch <= numEpochs; epoch++) {\n",
    "        // Iterate over dataset\n",
    "        System.out.print(\"Running epoch \" + epoch + \"...... \");\n",
    "        for (Batch batch : trainIter.getData(manager)) {\n",
    "\n",
    "                NDArray X = batch.getData().head();\n",
    "                NDArray y = batch.getLabels().head();\n",
    "\n",
    "                try (GradientCollector gc = Engine.getInstance().newGradientCollector()) {\n",
    "                    NDArray yHat = net(X, true); // net function call\n",
    "\n",
    "                    NDArray lossValue = loss.evaluate(new NDList(y), new NDList(yHat));\n",
    "                    NDArray l = lossValue.mul(batchSize);\n",
    "                    \n",
    "                    epochLoss += l.sum().getFloat();\n",
    "                    \n",
    "                    accuracyVal += Training.accuracy(yHat, y);\n",
    "                    gc.backward(l); // gradient calculation\n",
    "                }\n",
    "                \n",
    "                batch.close();\n",
    "                Training.sgd(params, lr, batchSize); // updater\n",
    "            }\n",
    "\n",
    "            trainLoss[epoch-1] = epochLoss/trainIter.size();\n",
    "            trainAccuracy[epoch-1] = accuracyVal/trainIter.size();\n",
    "            \n",
    "            epochLoss = 0f;\n",
    "            accuracyVal = 0f;\n",
    "\n",
    "            for (Batch batch : testIter.getData(manager)) {\n",
    "\n",
    "                NDArray X = batch.getData().head();\n",
    "                NDArray y = batch.getLabels().head();\n",
    "\n",
    "                NDArray yHat = net(X, false); // net function call\n",
    "                accuracyVal += Training.accuracy(yHat, y);\n",
    "            }\n",
    "\n",
    "            testAccuracy[epoch-1] = accuracyVal/testIter.size();\n",
    "            epochCount[epoch-1] = epoch;\n",
    "            accuracyVal = 0f;\n",
    "            System.out.println(\"Finished epoch \" + epoch);\n",
    "}\n",
    "\n",
    "System.out.println(\"Finished training!\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "String[] lossLabel = new String[trainLoss.length + testAccuracy.length + trainAccuracy.length];\n",
    "\n",
    "Arrays.fill(lossLabel, 0, trainLoss.length, \"train loss\");\n",
    "Arrays.fill(lossLabel, trainAccuracy.length, trainLoss.length + trainAccuracy.length, \"train acc\");\n",
    "Arrays.fill(lossLabel, trainLoss.length + trainAccuracy.length,\n",
    "                trainLoss.length + testAccuracy.length + trainAccuracy.length, \"test acc\");\n",
    "\n",
    "Table data = Table.create(\"Data\").addColumns(\n",
    "            DoubleColumn.create(\"epochCount\", ArrayUtils.addAll(epochCount, ArrayUtils.addAll(epochCount, epochCount))),\n",
    "            DoubleColumn.create(\"loss\", ArrayUtils.addAll(trainLoss, ArrayUtils.addAll(trainAccuracy, testAccuracy))),\n",
    "            StringColumn.create(\"lossLabel\", lossLabel)\n",
    ");\n",
    "\n",
    "render(LinePlot.create(\"\", data, \"epochCount\", \"loss\", \"lossLabel\"),\"text/html\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![lineplot](https://d2l-java-resources.s3.amazonaws.com/img/chapter_multilayer-perceptrons_dropout_output1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Concise Implementation\n",
    "\n",
    "Using DJL, all we need to do is add a `Dropout` layer\n",
    "after each fully-connected layer, \n",
    "passing in the dropout probability\n",
    "as the only argument to its constructor.\n",
    "During training, the `Dropout` layer will randomly\n",
    "drop out outputs of the previous layer\n",
    "(or equivalently, the inputs to the subsequent layer)\n",
    "according to the specified dropout probability.\n",
    "When the model is not in training mode,\n",
    "the `Dropout` layer simply passes the data through during testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "6"
    }
   },
   "outputs": [],
   "source": [
    "SequentialBlock net = new SequentialBlock();\n",
    "net.add(Blocks.batchFlattenBlock(784));\n",
    "net.add(Linear.builder().setUnits(256).build());\n",
    "net.add(Activation::relu);\n",
    "net.add(Dropout.builder().optRate(dropout1).build());\n",
    "net.add(Linear.builder().setUnits(256).build());\n",
    "net.add(Activation::relu);\n",
    "net.add(Dropout.builder().optRate(dropout2).build());\n",
    "net.add(Linear.builder().setUnits(10).build());\n",
    "net.setInitializer(new NormalInitializer(0.01f));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we train and test the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "7"
    }
   },
   "outputs": [],
   "source": [
    "Map<String, double[]> evaluatorMetrics = new HashMap<>();\n",
    "\n",
    "Tracker lrt = Tracker.fixed(0.5f);\n",
    "Optimizer sgd = Optimizer.sgd().setLearningRateTracker(lrt).build();\n",
    "\n",
    "Loss loss = Loss.softmaxCrossEntropyLoss();\n",
    "\n",
    "DefaultTrainingConfig config = new DefaultTrainingConfig(loss)\n",
    "                .optOptimizer(sgd) // Optimizer (loss function)\n",
    "                .optDevices(Device.getDevices(1)) // single GPU\n",
    "                .addEvaluator(new Accuracy()) // Model Accuracy\n",
    "                .addTrainingListeners(TrainingListener.Defaults.logging()); // Logging\n",
    "\n",
    "    try (Model model = Model.newInstance(\"mlp\")) {\n",
    "        model.setBlock(net);\n",
    "\n",
    "        try (Trainer trainer = model.newTrainer(config)) {\n",
    "\n",
    "            trainer.initialize(new Shape(1, 784));\n",
    "            trainer.setMetrics(new Metrics());\n",
    "\n",
    "            EasyTrain.fit(trainer, numEpochs, trainIter, testIter);\n",
    "\n",
    "            Metrics metrics = trainer.getMetrics();\n",
    "\n",
    "            trainer.getEvaluators().stream()\n",
    "                    .forEach(evaluator -> {\n",
    "                        evaluatorMetrics.put(\"train_epoch_\" + evaluator.getName(), metrics.getMetric(\"train_epoch_\" + evaluator.getName()).stream()\n",
    "                                            .mapToDouble(x -> x.getValue().doubleValue()).toArray());\n",
    "                        evaluatorMetrics.put(\"validate_epoch_\" + evaluator.getName(), metrics.getMetric(\"validate_epoch_\" + evaluator.getName()).stream()\n",
    "                                            .mapToDouble(x -> x.getValue().doubleValue()).toArray());\n",
    "            });\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainLoss = evaluatorMetrics.get(\"train_epoch_SoftmaxCrossEntropyLoss\");\n",
    "trainAccuracy = evaluatorMetrics.get(\"train_epoch_Accuracy\");\n",
    "testAccuracy = evaluatorMetrics.get(\"validate_epoch_Accuracy\");\n",
    "\n",
    "String[] lossLabel = new String[trainLoss.length + testAccuracy.length + trainAccuracy.length];\n",
    "\n",
    "Arrays.fill(lossLabel, 0, trainLoss.length, \"test acc\");\n",
    "Arrays.fill(lossLabel, trainAccuracy.length, trainLoss.length + trainAccuracy.length, \"train acc\");\n",
    "Arrays.fill(lossLabel, trainLoss.length + trainAccuracy.length,\n",
    "                trainLoss.length + testAccuracy.length + trainAccuracy.length, \"train loss\");\n",
    "\n",
    "Table data = Table.create(\"Data\").addColumns(\n",
    "            DoubleColumn.create(\"epochCount\", ArrayUtils.addAll(epochCount, ArrayUtils.addAll(epochCount, epochCount))),\n",
    "            DoubleColumn.create(\"loss\", ArrayUtils.addAll(testAccuracy , ArrayUtils.addAll(trainAccuracy, trainLoss))),\n",
    "            StringColumn.create(\"lossLabel\", lossLabel)\n",
    ");\n",
    "\n",
    "render(LinePlot.create(\"\", data, \"epochCount\", \"loss\", \"lossLabel\"),\"text/html\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![lineplot](https://d2l-java-resources.s3.amazonaws.com/img/chapter_multilayer-perceptrons_dropout_output2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "* Beyond controlling the number of dimensions and the size of the weight vector, dropout is yet another tool to avoid overfitting. Often all three are used jointly.\n",
    "* Dropout replaces an activation $h$ with a random variable $h'$ with expected value $h$ and with variance given by the dropout probability $p$.\n",
    "* Dropout is only used during training.\n",
    "\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. What happens if you change the dropout probabilities for layers 1 and 2? In particular, what happens if you switch the ones for both layers? Design an experiment to answer these questions, describe your results quantitatively, and summarize the qualitative takeaways.\n",
    "1. Increase the number of epochs and compare the results obtained when using dropout with those when not using it.\n",
    "1. What is the variance of the activations in each hidden layer when dropout is and is not applied? Draw a plot to show how this quantity evolves over time for both models. \n",
    "1. Why is dropout not typically used at test time?\n",
    "1. Using the model in this section as an example, compare the effects of using dropout and weight decay. What happens when dropout and weight decay are used at the same time? Are the results additive, are there diminished returns or (worse), do they cancel each other out?\n",
    "1. What happens if we apply dropout to the individual weights of the weight matrix rather than the activations?\n",
    "1. Invent another technique for injecting random noise at each layer that is different from the standard dropout technique. Can you develop a method that outperforms dropout on the Fashion-MNIST dataset (for a fixed architecture)?\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Java",
   "language": "java",
   "name": "java"
  },
  "language_info": {
   "codemirror_mode": "java",
   "file_extension": ".jshell",
   "mimetype": "text/x-java-source",
   "name": "Java",
   "pygments_lexer": "java",
   "version": "11.0.5+10-LTS"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
